In this project, We attempted to identify the lane boundaries in a video by building an advanced lane-finding algorithm using distortion correction, image rectification, color transforms, and gradient thresholding , identified lane curvature and vehicle displacement, overcame environmental challenges such as shadows and pavement changes.
The goals / steps of this project are the following:
Author : Tran Ly Vu
InΒ [1]:
import numpy as np
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import pickle
import numpy as np
import glob
import os
from moviepy.editor import VideoFileClip
from IPython.display import HTML
%matplotlib inline
The code for this step is contained in the 4th and 6th code cells of the IPython notebook located in "/notebook/advanced_lane_lines.ipynb" (or in lines 19 through 49 in /src/advanced_lane_lines.py).
I start by preparing "object points", which will be the (x, y, z) coordinates of the chessboard corners in the world. Here I am assuming the chessboard is fixed on the (x, y) plane at z=0, such that the object points are the same for each calibration image. Thus, objp
is just a replicated array of coordinates, and objpoints
will be appended with a copy of it every time I successfully detect all chessboard corners in a test image. imgpoints
will be appended with the (x, y) pixel position of each of the corners in the image plane with each successful chessboard detection.
I then used the output obj_points
and img_points
to compute the camera calibration and distortion coefficients using the cv2.calibrateCamera()
function. I applied this distortion correction to the test image using the cv2.undistort()
function and obtained this result:
InΒ [2]:
'''Read images directories'''
chessboard_images = glob.glob('../camera_cal/*.jpg')
test_images = glob.glob('../test_images/*.jpg')
#Prepare obj poinrts
objp = np.zeros((9 * 6, 3), np.float32)
objp[:,:2] = np.mgrid[0:9, 0:6].T.reshape(-1,2)
obj_points = [] #3d points in real world space
img_points = [] #2d points in image points
# termination criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
fig, axs = plt.subplots(5,4, figsize=(16, 11))
fig.subplots_adjust(hspace = .2, wspace=.001)
axs = axs.ravel()
for i, fname in enumerate(chessboard_images):
img = cv2.imread(fname)
image_name=os.path.split(fname)[1]
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
ret, corners = cv2.findChessboardCorners(gray, (9, 6), None)
if ret == True:
obj_points.append(objp)
# this step to refine image points was taken from:
# http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_calib3d/py_calibration/py_calibration.html
corners2 = cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)
img_points.append(corners2)
# Draw chessboard with corners
img = cv2.drawChessboardCorners(img, (9,6), corners, ret)
axs[i].axis('off')
axs[i].imshow(img)
InΒ [3]:
# Calcualte the undistortion matrix to calculate matrix and distance coefficient
img = cv2.imread(chessboard_images[0])
img_size = (img.shape[1], img.shape[0])
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(obj_points, img_points, img_size, None, None)
InΒ [4]:
def undistort_image(img):
result = cv2.undistort(img, mtx, dist, None, mtx)
return result
InΒ [5]:
## using 1 chessboard image from calibration directory
chessboard_img = cv2.imread(chessboard_images[0])
undistort_chessboard_img = undistort_image(chessboard_img)
# using 1 image from test image directory to build and demonstrate each step of the pipeline
bgr_test_img = cv2.imread(test_images[0])
rgb_test_img = cv2.cvtColor(bgr_test_img, cv2.COLOR_BGR2RGB)
undistort_test_img = undistort_image(rgb_test_img )
# Visualize undistortion
f, (ax1, ax2) = plt.subplots(2, 2, figsize=(20,10))
f.subplots_adjust(hspace = .2, wspace=.05)
ax1[0].imshow(chessboard_img)
ax1[0].set_title('Original chessboard image', fontsize=30)
ax1[1].imshow(undistort_chessboard_img)
ax1[1].set_title('undistorted chessboard image', fontsize=30)
ax2[0].imshow(rgb_test_img)
ax2[0].set_title('Original sample test image', fontsize=30)
ax2[1].imshow(undistort_test_img)
ax2[1].set_title('Undistorted sample test image', fontsize=30)
Out[5]:
I used a combination of color and gradient thresholds to filter out what we donβt want.
1. First , we do a color threshold filter to pick only yelow and white color of the road lanes
I first visualize 3 color spaces RGB, HLS and HSV (color space transformation is done using cv2). Among all, R-channel in RGB , S-channel in HLS and V-channel in HSV seem to work best in their respective space. Their difference is very insignificant. I eventually decided to use value-channel of HSV, although I believe others will do equavalently well.
With some error and trial by observation, I find the threshold (210, 255) to perform pretty well for V-channel in HSV.
InΒ [6]:
# Visualize multiple color space channels
example_img_unwarp_R = undistort_test_img[:,:,0]
example_img_unwarp_G = undistort_test_img[:,:,1]
example_img_unwarp_B = undistort_test_img[:,:,2]
example_img_unwarp_HLS = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2HLS)
example_img_unwarp_H = example_img_unwarp_HLS[:,:,0]
example_img_unwarp_L = example_img_unwarp_HLS[:,:,1]
example_img_unwarp_S = example_img_unwarp_HLS[:,:,2]
example_img_unwarp_HSV = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2HSV)
example_img_unwarp_H = example_img_unwarp_HSV[:,:,0]
example_img_unwarp_S = example_img_unwarp_HSV[:,:,1]
example_img_unwarp_V = example_img_unwarp_HSV[:,:,2]
fig, axs = plt.subplots(3,3, figsize=(30, 20))
fig.subplots_adjust(hspace = .01, wspace=.01)
axs = axs.ravel()
axs[0].imshow(example_img_unwarp_R, cmap='gray')
axs[0].set_title('RGB R-channel', fontsize=30)
axs[1].imshow(example_img_unwarp_G, cmap='gray')
axs[1].set_title('RGB G-Channel', fontsize=30)
axs[2].imshow(example_img_unwarp_B, cmap='gray')
axs[2].set_title('RGB B-channel', fontsize=30)
axs[3].imshow(example_img_unwarp_H, cmap='gray')
axs[3].set_title('HLS H-channel', fontsize=30)
axs[4].imshow(example_img_unwarp_L, cmap='gray')
axs[4].set_title('HLS L-Channel', fontsize=30)
axs[5].imshow(example_img_unwarp_S, cmap='gray')
axs[5].set_title('HLS S-Channel', fontsize=30)
axs[6].imshow(example_img_unwarp_H, cmap='gray')
axs[6].set_title('HSV H-channel', fontsize=30)
axs[7].imshow(example_img_unwarp_S, cmap='gray')
axs[7].set_title('HSV S-Channel', fontsize=30)
axs[8].imshow(example_img_unwarp_V, cmap='gray')
axs[8].set_title('HSV V-Channel', fontsize=30)
Out[6]:
InΒ [7]:
rgb_r_channel = undistort_test_img[:,:,0]
rgb_r_thresh = (200, 255)
rgb_r_binary = np.zeros_like(rgb_r_channel)
rgb_r_binary[(rgb_r_channel > rgb_r_thresh[0]) & (rgb_r_channel <= rgb_r_thresh[1])] = 1
hls = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2HLS)
hls_s_channel = hls[:,:,2]
hls_s_thresh = (150, 255)
hls_s_binary = np.zeros_like(hls_s_channel)
hls_s_binary[(hls_s_channel > hls_s_thresh[0]) & (hls_s_channel <= hls_s_thresh[1])] = 1
hsv = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2HSV)
hsv_v_channel = hsv[:,:,2]
hsv_v_thresh = (210, 255)
hsv_v_binary = np.zeros_like(hsv_v_channel)
hsv_v_binary[(hsv_v_channel > hsv_v_thresh[0]) & (hsv_v_channel <= hsv_v_thresh[1])] = 1
plt.imsave('../output_images/color_filter_binary.jpg', rgb_r_binary)
fig, axs = plt.subplots(2,2, figsize=(30, 20))
fig.subplots_adjust(hspace = .01, wspace=.01)
axs = axs.ravel()
axs[0].imshow(rgb_r_binary, cmap='gray')
axs[0].set_title('RGB r-binary', fontsize=30)
axs[1].imshow(hls_s_binary, cmap='gray')
axs[1].set_title('HLS s-binary', fontsize=30)
axs[2].imshow(hsv_v_binary, cmap='gray')
axs[2].set_title('HSV v-binary', fontsize=30)
Out[7]:
InΒ [8]:
def absolute_sobel_threshold(img, orient='x', thresh_min = 0, thresh_max = 255):
if orient == 'x':
abs_sobel = np.absolute(cv2.Sobel(img, cv2.CV_64F, 1, 0))
if orient == 'y':
abs_sobel = np.absolute(cv2.Sobel(img, cv2.CV_64F, 0, 1))
scaled_sobel = np.uint8(255 * abs_sobel/np.max(abs_sobel))
sobel_binary = np.zeros_like(scaled_sobel)
sobel_binary[(scaled_sobel >= thresh_min) & (scaled_sobel <= thresh_max)] = 1
return sobel_binary
InΒ [9]:
gray_img = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2GRAY)
grad_x_binary = absolute_sobel_threshold(gray_img , orient='x', thresh_min=60, thresh_max=150)
grad_y_binary = absolute_sobel_threshold(gray_img , orient='y', thresh_min=90, thresh_max=255)
#Plot the result
fig, axs = plt.subplots(1,2, figsize=(24, 9))
fig.subplots_adjust(hspace = .01, wspace=.01)
axs = axs.ravel()
axs[0].imshow(grad_x_binary, cmap='gray')
axs[0].set_title('gradient x binary', fontsize=30)
axs[1].imshow(grad_y_binary, cmap='gray')
axs[1].set_title('gradient y binary', fontsize=30)
Out[9]:
InΒ [10]:
def magnitude_threshold(gray_img, sobel_kernel = 3, mag_thresh = (0, 255)):
sobelx = cv2.Sobel(gray_img, cv2.CV_64F, 1, 0, ksize = sobel_kernel)
sobely = cv2.Sobel(gray_img, cv2.CV_64F, 0, 1, ksize = sobel_kernel)
grad_mag = np.sqrt(sobelx**2 + sobely**2)
scale_factor = np.max(grad_mag)/255
grad_mag = (grad_mag/scale_factor).astype(np.uint8)
mag_binary = np.zeros_like(grad_mag)
mag_binary[(grad_mag >= mag_thresh[0]) & (grad_mag <= mag_thresh[1])] = 1
return mag_binary
InΒ [11]:
gray_img = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2GRAY)
mag_binary = magnitude_threshold(gray_img, sobel_kernel = 31, mag_thresh = (135, 255))
#Plot the result
fig, axs = plt.subplots(1,2, figsize=(24, 9))
fig.subplots_adjust(hspace = .01, wspace=.01)
axs = axs.ravel()
axs[0].imshow(undistort_test_img, cmap='gray')
axs[0].set_title('original image', fontsize=30)
axs[1].imshow(mag_binary , cmap='gray')
axs[1].set_title('thresholded gradient magnitude', fontsize=30)
Out[11]:
InΒ [12]:
def direction_threshold(gray_img, sobel_kernel = 3, dir_thresh = (0, np.pi/2)):
sobelx = cv2.Sobel(gray_img, cv2.CV_64F, 1, 0, ksize = sobel_kernel)
sobely = cv2.Sobel(gray_img, cv2.CV_64F, 0, 1, ksize = sobel_kernel)
abs_grad_dir = np.arctan2(np.absolute(sobely), np.absolute(sobelx))
dir_binary = np.zeros_like(abs_grad_dir)
dir_binary[(abs_grad_dir >= dir_thresh[0]) & (abs_grad_dir <= dir_thresh[1])] = 1
return dir_binary
InΒ [13]:
gray_img = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2GRAY)
dir_binary = direction_threshold(gray_img, sobel_kernel = 23, dir_thresh = (0.7, 1.1))
#Plot the result
fig, axs = plt.subplots(1,2, figsize=(24, 9))
fig.subplots_adjust(hspace = .01, wspace=.01)
axs = axs.ravel()
axs[0].imshow(undistort_test_img, cmap='gray')
axs[0].set_title('original image', fontsize = 30)
axs[1].imshow(dir_binary , cmap='gray')
axs[1].set_title('thresholded gradient direction', fontsize = 30)
Out[13]:
InΒ [14]:
gray_img = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2GRAY)
grad_x_binary = absolute_sobel_threshold(gray_img, orient='x', thresh_min=60, thresh_max=150)
grad_y_binary = absolute_sobel_threshold(gray_img, orient='y', thresh_min=90, thresh_max=255)
mag_binary = magnitude_threshold(gray_img, sobel_kernel = 31, mag_thresh = (135, 255))
#dir_binary = direction_threshold(gray_img, sobel_kernel = 23, dir_thresh = (0.7, 1.1))
combined = np.zeros_like(gray_img)
combined[(grad_x_binary == 1) & (grad_y_binary == 1) & (mag_binary == 1) ] = 1
#Plot the result
fig, axs = plt.subplots(1,2, figsize=(24, 9))
fig.subplots_adjust(hspace = .01, wspace=.01)
axs = axs.ravel()
axs[0].imshow(undistort_test_img, cmap='gray')
axs[0].set_title('original image', fontsize=30)
axs[1].imshow(combined, cmap='gray')
axs[1].set_title('thresholded combine gradient', fontsize=30)
Out[14]:
InΒ [15]:
gray_img = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2GRAY)
grad_x_binary = absolute_sobel_threshold(gray_img, orient='x', thresh_min=30, thresh_max=250)
grad_y_binary = absolute_sobel_threshold(gray_img, orient='y', thresh_min=20, thresh_max=100)
mag_binary = magnitude_threshold(gray_img, sobel_kernel = 9, mag_thresh = (120, 255))
dir_binary = direction_threshold(gray_img, sobel_kernel = 15, dir_thresh = (0.7, 1.1))
combined_threshold = np.zeros_like(dir_binary)
combined_threshold[((grad_x_binary == 1) & (grad_y_binary == 1)) | ((mag_binary == 1) & (dir_binary == 1))] = 1
hsv = cv2.cvtColor(undistort_test_img, cv2.COLOR_RGB2HSV)
hsv_v_channel = hsv[:,:,2]
hsv_v_thresh = (210, 255)
hsv_v_binary = np.zeros_like(hsv_v_channel)
hsv_v_binary[(hsv_v_channel > hsv_v_thresh[0]) & (hsv_v_channel <= hsv_v_thresh[1])] = 1
#rgb_r_channel = undistort_test_img[:,:,0]
#rgb_r_thresh = (200, 255)
#rgb_r_binary = np.zeros_like(rgb_r_channel)
#rgb_r_binary[(rgb_r_channel > rgb_r_thresh[0]) & (rgb_r_channel <= rgb_r_thresh[1])] = 1
# Stack each channel to view their individual contributions in green and blue respectively
# This returns a stack of the two binary images, whose components you can see as different colors
color_binary = np.dstack((combined_threshold, combined_threshold, hsv_v_binary)) * 255
# Combine the two binary thresholds
combined_binary = np.zeros_like(hsv_v_channel)
combined_binary[(rgb_r_binary == 1) | (combined_threshold == 1)] = 1
plt.imsave('../output_images/combine_binary.jpg', combined_binary)
#Plot the result
fig, axs = plt.subplots(1,2, figsize=(24, 9))
fig.subplots_adjust(hspace = .01, wspace=.01)
axs = axs.ravel()
axs[0].imshow(undistort_test_img, cmap='gray')
axs[0].set_title('original image', fontsize = 30)
axs[1].imshow(combined_binary, cmap='gray')
axs[1].set_title('Thresholded combined gradient and color', fontsize = 30)
Out[15]:
The code for my perspective transform includes a function called warper()
, which appears in lines 1 through 8 in the file src/advanced_lane_lines.py
. The warper()
function takes as inputs an image (img
), as well as source (src
) and destination (dst
) points. I chose the hardcode the source and destination points in the following manner:
src = np.float32(
[[(img_size[0] / 2) - 55, img_size[1] / 2 + 100],
[((img_size[0] / 6) - 10), img_size[1]],
[(img_size[0] * 5 / 6) + 60, img_size[1]],
[(img_size[0] / 2 + 55), img_size[1] / 2 + 100]])
dst = np.float32([
[(img_size[0] / 4), 0],
[(img_size[0] / 4), img_size[1]],
[(img_size[0] * 3 / 4), img_size[1]],
[(img_size[0] * 3 / 4), 0]
])
This resulted in the following source and destination points:
Source | Destination |
---|---|
585, 460 | 320, 0 |
203, 720 | 320, 720 |
1127, 720 | 960, 720 |
695, 460 | 960, 0 |
InΒ [16]:
def warper(image, source, destination):
h,w = img.shape[:2]
transform_matrix = cv2.getPerspectiveTransform(source, destination)
inverse_matrix = cv2.getPerspectiveTransform(destination, source)
warped = cv2.warpPerspective(image, transform_matrix , (w,h))
return warped, transform_matrix, inverse_matrix
def source(img_size):
src = np.float32(
[
[(img_size[0] / 2) - 55, img_size[1] / 2 + 100],
[(img_size[0] / 2 + 55), img_size[1] / 2 + 100],
[((img_size[0] / 6) - 10), img_size[1]],
[(img_size[0] * 5 / 6) + 60, img_size[1]]
])
return src
def destination(img_size):
dest = np.float32(
[
[(img_size[0] / 4), 0],
[(img_size[0] * 3 / 4), 0],
[(img_size[0] / 4), img_size[1]],
[(img_size[0] * 3 / 4), img_size[1]]
])
return dest
InΒ [17]:
img_size = (undistort_test_img.shape[1], undistort_test_img.shape[0])
print(img_size)
src = source(img_size)
dest = destination(img_size)
warp_example_img, M, inverse_M = warper(undistort_test_img, src, dest)
plt.imsave('../output_images/warp_example_img.jpg', warp_example_img)
# Visualize unwarp
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,10))
f.subplots_adjust(hspace = .2, wspace=.05)
ax1.imshow(undistort_test_img)
x = [src[0][0],src[2][0],src[3][0],src[1][0],src[0][0]]
y = [src[0][1],src[2][1],src[3][1],src[1][1],src[0][1]]
ax1.plot(x, y, color='#33cc99', alpha=0.4, linewidth=3, solid_capstyle='round', zorder=2)
ax1.set_title('Undistorted Image', fontsize=30)
ax2.imshow(warp_example_img)
ax2.set_title('Unwarped Image', fontsize=30)
Out[17]:
InΒ [18]:
def pipeline(img):
img_size = (img.shape[1], img.shape[0])
src = source(img_size)
dest = destination(img_size)
warp_img, M, inverse_M = warper(img, src, dest)
gray_img = cv2.cvtColor(warp_img, cv2.COLOR_RGB2GRAY)
grad_x_binary = absolute_sobel_threshold(gray_img, orient='x', thresh_min=30, thresh_max=250)
grad_y_binary = absolute_sobel_threshold(gray_img, orient='y', thresh_min=20, thresh_max=100)
mag_binary = magnitude_threshold(gray_img, sobel_kernel = 9, mag_thresh = (120, 255))
dir_binary = direction_threshold(gray_img, sobel_kernel = 15, dir_thresh = (0.7, 1.1))
combined_threshold = np.zeros_like(dir_binary)
combined_threshold[((grad_x_binary == 1) & (grad_y_binary == 1)) & (mag_binary == 1) ] = 1
hsv = cv2.cvtColor(warp_img, cv2.COLOR_RGB2HSV)
hsv_v_channel = hsv[:,:,2]
hsv_v_thresh = (210, 255)
hsv_v_binary = np.zeros_like(hsv_v_channel)
hsv_v_binary[(hsv_v_channel > hsv_v_thresh[0]) & (hsv_v_channel <= hsv_v_thresh[1])] = 1
# Stack each channel to view their individual contributions in green and blue respectively
# This returns a stack of the two binary images, whose components you can see as different colors
color_binary = np.dstack((combined_threshold, combined_threshold, hsv_v_binary)) * 255
# Combine the two binary thresholds
combined_binary = np.zeros_like(gray_img)
combined_binary[(hsv_v_binary == 1) | (combined_threshold == 1)] = 1
return combined_binary, inverse_M
InΒ [19]:
bgr_test_img_2 = cv2.imread(test_images[0])
rgb_test_img_2 = cv2.cvtColor(bgr_test_img_2, cv2.COLOR_BGR2RGB)
undistort_test_img = undistort_image(rgb_test_img)
binary_warped, inverse_M = pipeline(undistort_test_img)
fig, axs = plt.subplots(1,1, figsize=(24, 9))
axs.imshow(binary_warped, cmap='gray')
axs.set_title('combined binary', fontsize = 30)
Out[19]:
I used second order polynomial to fit the lane: x = ay**2 + by + c.
In order to better estimate where the lane is, we use a histogram on the bottom half of image
Then we divide the image in windows, and for each left and right window we find the mean of it, re-centering the window. The points inside the windows are stored. We then feed the numpy polyfit function to find the best second order polynomial to represent the lanes.
InΒ [20]:
# Take a histogram of the bottom half of the image
histogram = np.sum(binary_warped[binary_warped.shape[0]//2:,:], axis=0)
plt.title('Histogram')
plt.xlabel("Fixed Position")
plt.ylabel("Counts")
plt.plot(histogram)
plt.savefig('../output_images/histogram.jpg')
InΒ [21]:
#def test_polyfit
out_img = np.dstack((binary_warped, binary_warped, binary_warped)) * 255
# Take a histogram of the bottom half of the image
histogram = np.sum(binary_warped[binary_warped.shape[0]//2:,:], axis=0)
midpoint = np.int(histogram.shape[0]/2)
leftx_base = np.argmax(histogram[:midpoint])
rightx_base = np.argmax(histogram[midpoint:]) + midpoint
# Choose the number of sliding windows
nwindows = 10
# Set height of windows
window_height = np.int(binary_warped.shape[0]/ nwindows)
# Identify the x and y positions of all nonzero pixels in the image
nonzero = binary_warped.nonzero()
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
# Current positions to be updated for each window
leftx_current = leftx_base
rightx_current = rightx_base
# Set the width of the windows +/- margin
margin = 100
# Set minimum number of pixels found to recenter window
minpix = 50
# Create empty lists to receive left and right lane pixel indices
left_lane_inds = []
right_lane_inds = []
# Step through the windows one by one
for window in range(nwindows):
# Identify window boundaries in x and y (and right and left)
win_y_low = binary_warped.shape[0] - (window + 1) * window_height
win_y_high = binary_warped.shape[0] - window * window_height
win_xleft_low = leftx_current - margin
win_xleft_high = leftx_current + margin
win_xright_low = rightx_current - margin
win_xright_high = rightx_current + margin
# Draw the windows on the visualization image
cv2.rectangle(out_img,(win_xleft_low,win_y_low),(win_xleft_high,win_y_high),(0,255,0), 2)
cv2.rectangle(out_img,(win_xright_low,win_y_low),(win_xright_high,win_y_high),(0,255,0), 2)
# Identify the nonzero pixels in x and y within the window
good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & (nonzerox >= win_xleft_low) & (nonzerox < win_xleft_high)).nonzero()[0]
good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & (nonzerox >= win_xright_low) & (nonzerox < win_xright_high)).nonzero()[0]
# Append these indices to the lists
left_lane_inds.append(good_left_inds)
right_lane_inds.append(good_right_inds)
# If you found > minpix pixels, recenter next window on their mean position
if len(good_left_inds) > minpix:
leftx_current = np.int(np.mean(nonzerox[good_left_inds]))
if len(good_right_inds) > minpix:
rightx_current = np.int(np.mean(nonzerox[good_right_inds]))
# Concatenate the arrays of indices
left_lane_inds = np.concatenate(left_lane_inds)
right_lane_inds = np.concatenate(right_lane_inds)
# Extract left and right line pixel positions
leftx = nonzerox[left_lane_inds]
lefty = nonzeroy[left_lane_inds]
rightx = nonzerox[right_lane_inds]
righty = nonzeroy[right_lane_inds]
# Fit a second order polynomial to each
left_fit = np.polyfit(lefty, leftx, 2)
right_fit = np.polyfit(righty, rightx, 2)
# Generate x and y values for plotting
ploty = np.linspace(0, binary_warped.shape[0]-1, binary_warped.shape[0] )
left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]
out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0]
out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255]
#Plot the result
plt.imshow(out_img)
plt.plot(left_fitx, ploty, color='yellow')
plt.plot(right_fitx, ploty, color='yellow')
plt.xlim(0, 1280)
plt.ylim(720, 0)
Out[21]:
InΒ [22]:
# Assume you now have a new warped binary image
# from the next frame of video (also called "binary_warped")
# It's now much easier to find line pixels!
nonzero = binary_warped.nonzero()
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
margin = 100
left_lane_inds = ((nonzerox > (left_fit[0]*(nonzeroy**2) + left_fit[1]*nonzeroy +
left_fit[2] - margin)) & (nonzerox < (left_fit[0]*(nonzeroy**2) +
left_fit[1]*nonzeroy + left_fit[2] + margin)))
right_lane_inds = ((nonzerox > (right_fit[0]*(nonzeroy**2) + right_fit[1]*nonzeroy +
right_fit[2] - margin)) & (nonzerox < (right_fit[0]*(nonzeroy**2) +
right_fit[1]*nonzeroy + right_fit[2] + margin)))
# Again, extract left and right line pixel positions
leftx = nonzerox[left_lane_inds]
lefty = nonzeroy[left_lane_inds]
rightx = nonzerox[right_lane_inds]
righty = nonzeroy[right_lane_inds]
# Fit a second order polynomial to each
left_fit_update = np.polyfit(lefty, leftx, 2)
right_fit_update = np.polyfit(righty, rightx, 2)
# Visualization
margin = 100
# Generate x and y values for plotting
ploty = np.linspace(0, binary_warped.shape[0]-1, binary_warped.shape[0] )
left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]
# Create an image to draw on and an image to show the selection window
out_img = np.dstack((binary_warped, binary_warped, binary_warped))*255
window_img = np.zeros_like(out_img)
# Color in left and right line pixels
out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0]
out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255]
# Generate a polygon to illustrate the search window area
# And recast the x and y points into usable format for cv2.fillPoly()
left_line_window1 = np.array([np.transpose(np.vstack([left_fitx-margin, ploty]))])
left_line_window2 = np.array([np.flipud(np.transpose(np.vstack([left_fitx+margin, ploty])))])
left_line_pts = np.hstack((left_line_window1, left_line_window2))
right_line_window1 = np.array([np.transpose(np.vstack([right_fitx-margin, ploty]))])
right_line_window2 = np.array([np.flipud(np.transpose(np.vstack([right_fitx+margin, ploty])))])
right_line_pts = np.hstack((right_line_window1, right_line_window2))
# Draw the lane onto the warped blank image
cv2.fillPoly(window_img, np.int_([left_line_pts]), (0,255, 0))
cv2.fillPoly(window_img, np.int_([right_line_pts]), (0,255, 0))
result = cv2.addWeighted(out_img, 1, window_img, 0.3, 0)
plt.imshow(result)
plt.plot(left_fitx, ploty, color='yellow')
plt.plot(right_fitx, ploty, color='yellow')
plt.xlim(0, 1280)
plt.ylim(720, 0)
Out[22]:
Radius of curvature is implemented from this tutorial. For second order polynomials, it is simplified to (1+(2Ay + B)2)1.5/ abs(2A).
I calculated both radius curvature in pixel and real world spaces, for example, here is radius curvature of the example image:
Pixel space: left curvature: 4826.07068768 , right curvature: 2966.83010737
World space: left curvature: 2003.07070208 m , right curvature: 1745.93214794 m
InΒ [46]:
# fit = [A,B,C] , radisu in pixel space
def calculate_curvature_pixel_space(fit, ploty):
y_value = np.max(ploty)
nominator = (1 + (2 * fit[0] * y_value + fit[1])**2)**1.5
denominator = np.absolute(2 * fit[0])
return (nominator / denominator)
def calculate_curvature_world_space(fit, ploty):
# Define conversions in x and y from pixels space to meters
y_in_m_per_pix = 30/720 # meters per pixel in y dimension
x_in_m_per_pix = 3.7/700 # meters per pixel in x dimension
# Fit new polynomials to x,y in world space
fit_x = fit[0]*ploty**2 + fit[1] * ploty + fit[2]
fit_word_space = np.polyfit(ploty * y_in_m_per_pix, fit_x* x_in_m_per_pix, 2)
result = calculate_curvature_pixel_space(fit_word_space, ploty)
return result
# choose the maximum y-value, corresponding to the bottom of the image
left_curvature_pixel_space = calculate_curvature_pixel_space(left_fit, ploty)
right_curvature_pixel_space = calculate_curvature_pixel_space(right_fit, ploty)
left_curvature_world_space = calculate_curvature_world_space(left_fit, ploty)
right_curvature_world_space = calculate_curvature_world_space(right_fit, ploty)
print("Pixel space: left curvature: " + str(left_curvature_pixel_space) + ' , right curvature: ' + str(right_curvature_pixel_space))
print("World space: left curvature: " + str(left_curvature_world_space) + ' m , right curvature: ' + str(right_curvature_world_space) + ' m')
# compute the offset from the center\
xm_per_pix = 3.7/700 # meters per pixel in x dimension
ym_per_pix = 30/720
# Fit a second order polynomial to each
left_fit_m = np.polyfit(ploty*ym_per_pix, left_fitx*xm_per_pix, 2)
right_fit_m = np.polyfit(ploty*ym_per_pix, right_fitx*xm_per_pix, 2)
xMax = img.shape[1]*xm_per_pix
yMax = img.shape[0]*ym_per_pix
vehicleCenter = xMax / 2
lineLeft = left_fit_m[0]*yMax**2 + left_fit_m[1]*yMax + left_fit_m[2]
lineRight = right_fit_m[0]*yMax**2 + right_fit_m[1]*yMax + right_fit_m[2]
lineMiddle = lineLeft + (lineRight - lineLeft)/2
diffFromVehicle = lineMiddle - vehicleCenter
offset_string = "Center offset: %.2f m" % diffFromVehicle
print(offset_string)
InΒ [53]:
def drawing(undistorted_img, binary_warped, left_fit, right_fit, inverse_M, img_size):
# Create an image to draw the lines on
ploty = np.linspace(0, binary_warped.shape[0] - 1, binary_warped.shape[0] )
left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]
warp_zero = np.zeros_like(binary_warped).astype(np.uint8)
color_warp = np.dstack((warp_zero, warp_zero, warp_zero))
# Recast the x and y points into usable format for cv2.fillPoly()
pts_left = np.array([np.transpose(np.vstack([left_fitx, ploty]))])
pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, ploty])))])
pts = np.hstack((pts_left, pts_right))
# Draw the lane onto the warped blank image
cv2.fillPoly(color_warp, np.int_([pts]), (0,255, 0))
# Warp the blank back to original image space using inverse perspective matrix (Minv)
newwarp = cv2.warpPerspective(color_warp, inverse_M, img_size)
# Combine the result with the original image
result = cv2.addWeighted(undistorted_img, 1, newwarp, 0.3, 0)
left_curvature_world_space = calculate_curvature_world_space(left_fit, ploty)
right_curvature_world_space = calculate_curvature_world_space(right_fit, ploty)
curvature_string = "Left Radius of curvature: %.2f m" % left_curvature_world_space
cv2.putText(result,curvature_string , (100, 90), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (255,255,255), thickness=2)
curvature_string = "Right Radius of curvature: %.2f m" % right_curvature_world_space
cv2.putText(result,curvature_string , (100, 150), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (255,255,255), thickness=2)
# compute the offset from the center\
xm_per_pix = 3.7/700 # meters per pixel in x dimension
ym_per_pix = 30/720
# Fit a second order polynomial to each
left_fit_m = np.polyfit(ploty*ym_per_pix, left_fitx*xm_per_pix, 2)
right_fit_m = np.polyfit(ploty*ym_per_pix, right_fitx*xm_per_pix, 2)
xMax = img.shape[1]*xm_per_pix
yMax = img.shape[0]*ym_per_pix
vehicleCenter = xMax / 2
lineLeft = left_fit_m[0]*yMax**2 + left_fit_m[1]*yMax + left_fit_m[2]
lineRight = right_fit_m[0]*yMax**2 + right_fit_m[1]*yMax + right_fit_m[2]
lineMiddle = lineLeft + (lineRight - lineLeft)/2
diffFromVehicle = lineMiddle - vehicleCenter
offset_string = "Center offset: %.2f m" % diffFromVehicle
cv2.putText(result, offset_string, (100, 210), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (255,255,255), thickness=2)
return result
result = drawing(rgb_test_img, binary_warped, left_fit, right_fit, inverse_M, img_size)
plt.imsave("../output_images/test_img_1_output.jpg", result)
InΒ [54]:
def sliding_window_polyfit(binary_warped):
out_img = np.dstack((binary_warped, binary_warped, binary_warped)) * 255
# Take a histogram of the bottom half of the image
histogram = np.sum(binary_warped[binary_warped.shape[0]//2:,:], axis=0)
## fiding left and right point
midpoint = np.int(histogram.shape[0]/2)
leftx_base = np.argmax(histogram[:midpoint])
rightx_base = np.argmax(histogram[midpoint:]) + midpoint
# Choose the number of sliding windows
nwindows = 10
# Set height of windows
window_height = np.int(binary_warped.shape[0]/ nwindows)
# Identify the x and y positions of all nonzero pixels in the image
nonzero = binary_warped.nonzero()
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
# Set the width of the windows +/- margin
margin = 100
# Set minimum number of pixels found to recenter window
minpix = 50
# Create empty lists to receive left and right lane pixel indices
left_lane_inds = []
right_lane_inds = []
# Step through the windows one by one
for window in range(nwindows):
# Identify window boundaries in x and y (and right and left)
win_y_low = binary_warped.shape[0] - (window + 1) * window_height
win_y_high = binary_warped.shape[0] - window * window_height
win_xleft_low = leftx_base - margin
win_xleft_high = leftx_base + margin
win_xright_low = rightx_base - margin
win_xright_high = rightx_base + margin
# Draw the windows on the visualization image
cv2.rectangle(out_img,(win_xleft_low,win_y_low),(win_xleft_high,win_y_high),(0,255,0), 2)
cv2.rectangle(out_img,(win_xright_low,win_y_low),(win_xright_high,win_y_high),(0,255,0), 2)
# Identify the nonzero pixels in x and y within the window
good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & (nonzerox >= win_xleft_low) &
(nonzerox < win_xleft_high)).nonzero()[0]
good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & (nonzerox >= win_xright_low) &
(nonzerox < win_xright_high)).nonzero()[0]
# Append these indices to the lists
left_lane_inds.append(good_left_inds)
right_lane_inds.append(good_right_inds)
# If you found > minpix pixels, recenter next window on their mean position
if len(good_left_inds) > minpix:
leftx_current = np.int(np.mean(nonzerox[good_left_inds]))
if len(good_right_inds) > minpix:
rightx_current = np.int(np.mean(nonzerox[good_right_inds]))
# Concatenate the arrays of indices
left_lane_inds = np.concatenate(left_lane_inds)
right_lane_inds = np.concatenate(right_lane_inds)
# Extract left and right line pixel positions
leftx = nonzerox[left_lane_inds]
lefty = nonzeroy[left_lane_inds]
rightx = nonzerox[right_lane_inds]
righty = nonzeroy[right_lane_inds]
# Fit a second order polynomial to each
polyfit_left = np.polyfit(lefty, leftx, 2)
polyfit_right = np.polyfit(righty, rightx, 2)
return polyfit_left, polyfit_right
def polyfit_using_previous_fit(binary_warped, prev_left_fit, prev_right_fit):
# Assume you now have a new warped binary image
# from the next frame of video (also called "binary_warped")
# It's now much easier to find line pixels!
nonzero = binary_warped.nonzero()
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
margin = 100
left_lane_inds = ((nonzerox > (prev_left_fit[0]*(nonzeroy**2) + prev_left_fit[1]*nonzeroy + prev_left_fit[2] - margin))
& (nonzerox < (prev_left_fit[0]*(nonzeroy**2) + prev_left_fit[1]*nonzeroy + prev_left_fit[2] + margin)))
right_lane_inds = ((nonzerox > (prev_right_fit[0]*(nonzeroy**2) + prev_right_fit[1]*nonzeroy + prev_right_fit[2]- margin))
& (nonzerox < (prev_right_fit[0]*(nonzeroy**2) + prev_right_fit[1]*nonzeroy + prev_right_fit[2] + margin)))
# Again, extract left and right line pixel positions
leftx = nonzerox[left_lane_inds]
lefty = nonzeroy[left_lane_inds]
rightx = nonzerox[right_lane_inds]
righty = nonzeroy[right_lane_inds]
# Fit a second order polynomial to each
if len(leftx) != 0:
polyfit_left = np.polyfit(lefty, leftx, 2)
if len(rightx) != 0:
polyfit_right = np.polyfit(righty, rightx, 2)
return polyfit_left, polyfit_right
def process_image(image):
# global variables to store the polynomial coefficients of the line detected in the last frame
global polyfit_left
global polyfit_right
img_size = (image.shape[1], image.shape[0])
#1. Camera correction and distortion correction
undistorted_img = undistort_image(image)
#2. Applying perspective transformation, thresholded color and gradient filter
binary_warped, inverse_M = pipeline(undistorted_img)
#3. Detect lanes and return fit curves
if (polyfit_left is not None) and (polyfit_right is not None):
polyfit_left, polyfit_right = polyfit_using_previous_fit(binary_warped, polyfit_left, polyfit_right)
else:
polyfit_left, polyfit_right = sliding_window_polyfit(binary_warped)
result = drawing(undistorted_img, binary_warped, polyfit_left, polyfit_right, inverse_M, img_size)
return result
InΒ [55]:
polyfit_left = None
polyfit_right = None
clip1 = VideoFileClip('../test_video/project_video.mp4')
video_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
video_output = '../output_videos/project_video.mp4'
%time video_clip.write_videofile(video_output, audio=False)
The video is uploaded to here
InΒ [56]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</videot
""".format(video_output))
Out[56]:
InΒ [Β ]: